{"id":401904,"date":"2025-12-30T11:10:32","date_gmt":"2025-12-30T18:10:32","guid":{"rendered":"https:\/\/barrazacarlos.com\/free-robots-txt-generator\/"},"modified":"2025-12-30T11:10:35","modified_gmt":"2025-12-30T18:10:35","slug":"free-robots-txt-generator","status":"publish","type":"page","link":"https:\/\/barrazacarlos.com\/id\/free-robots-txt-generator\/","title":{"rendered":"Generator Robots.txt Gratis (Aman untuk SEO + Kontrol Perayap AI)"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-page\" data-elementor-id=\"401904\" class=\"elementor elementor-401904 elementor-400576\" data-elementor-post-type=\"page\">\n\t\t\t\t\t\t<main class=\"elementor-section elementor-top-section elementor-element elementor-element-23bfe264 elementor-section-height-min-height elementor-section-content-bottom elementor-section-boxed elementor-section-height-default elementor-section-items-middle\" data-id=\"23bfe264\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t\t<div class=\"elementor-background-overlay\"><\/div>\n\t\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-wider\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-619cb1bc\" data-id=\"619cb1bc\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-564894ed elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"564894ed\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-23e29665\" data-id=\"23e29665\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-1f79b8e3 elementor-widget elementor-widget-heading\" data-id=\"1f79b8e3\" data-element_type=\"widget\" data-e-type=\"widget\" data-settings=\"{&quot;_animation&quot;:&quot;none&quot;,&quot;_animation_delay&quot;:&quot;800&quot;}\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h1 class=\"elementor-heading-title elementor-size-default\">Generator Robots.txt Gratis (Aman untuk SEO + Kontrol Perayap AI)<\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-70a5da2c elementor-widget elementor-widget-heading\" data-id=\"70a5da2c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<span class=\"elementor-heading-title elementor-size-default\">Ditulis oleh<\/span>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-642f6018 elementor-align-left elementor-widget elementor-widget-post-info\" data-id=\"642f6018\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"post-info.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<ul class=\"elementor-inline-items elementor-icon-list-items elementor-post-info\">\n\t\t\t\t\t\t\t\t<li class=\"elementor-icon-list-item elementor-repeater-item-ef43aea elementor-inline-item\" itemprop=\"author\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-icon-list-text elementor-post-info__item elementor-post-info__item--type-author\">\n\t\t\t\t\t\t\t\t\t\tRedaction Team\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t<li class=\"elementor-icon-list-item elementor-repeater-item-476dca5 elementor-inline-item\" itemprop=\"datePublished\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t<span class=\"elementor-icon-list-text elementor-post-info__item elementor-post-info__item--type-date\">\n\t\t\t\t\t\t\t\t\t\t<time>Desember 30, 2025<\/time>\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t<\/ul>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5c576898 elementor-widget elementor-widget-text-editor\" data-id=\"5c576898\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p data-start=\"371\" data-end=\"608\">Gunakan <strong data-start=\"379\" data-end=\"408\">generator robots.txt gratis<\/strong> kami <strong data-start=\"379\" data-end=\"408\">untuk membuat<\/strong> <strong data-start=\"412\" data-end=\"450\">file robots.txt secara instan yang membantu Google<\/strong> dan <strong data-start=\"462\" data-end=\"536\">mesin pencari<\/strong> lainnya <strong data-start=\"462\" data-end=\"536\">merayapi dan mengindeks situs web Anda dengan benar, tanpa membuang-buang<\/strong> anggaran perayapan atau mengekspos bagian sensitif situs Anda.<\/p><p data-start=\"610\" data-end=\"825\"><strong data-start=\"615\" data-end=\"635\">Alat yang mudah digunakan<\/strong> ini <strong data-start=\"615\" data-end=\"635\">menghasilkan<\/strong> <strong data-start=\"646\" data-end=\"693\">file robots.txt yang efektif untuk situs web Anda, mengikuti<\/strong> <strong data-start=\"709\" data-end=\"794\">protokol pengecualian<\/strong> robot <strong data-start=\"709\" data-end=\"794\">(juga disebut protokol pengecualian robot atau standar) dan praktik terbaik SEO modern<\/strong>.<\/p><p data-start=\"827\" data-end=\"993\">Apakah Anda ingin <strong data-start=\"847\" data-end=\"938\">mengizinkan atau melarang perayap mengakses URL, folder, direktori, atau halaman tertentu,<\/strong> <strong data-start=\"945\" data-end=\"969\">generator<\/strong> txt <strong data-start=\"847\" data-end=\"938\">robot ini<\/strong> <strong data-start=\"945\" data-end=\"969\">membantu Anda melakukannya dengan aman<\/strong>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t<div class=\"elementor-column elementor-col-50 elementor-inner-column elementor-element elementor-element-761c4a6c\" data-id=\"761c4a6c\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-2adebe2e elementor-widget elementor-widget-image\" data-id=\"2adebe2e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img fetchpriority=\"high\" decoding=\"async\" width=\"1280\" height=\"720\" src=\"https:\/\/barrazacarlos.com\/wp-content\/uploads\/2025\/12\/Free-Robots.txt-Generator.webp\" class=\"attachment-full size-full wp-image-400583\" alt=\"Free Robots.txt Generator\" srcset=\"https:\/\/barrazacarlos.com\/wp-content\/uploads\/2025\/12\/Free-Robots.txt-Generator.webp 1280w, https:\/\/barrazacarlos.com\/wp-content\/uploads\/2025\/12\/Free-Robots.txt-Generator-300x169.webp 300w, https:\/\/barrazacarlos.com\/wp-content\/uploads\/2025\/12\/Free-Robots.txt-Generator-1024x576.webp 1024w, https:\/\/barrazacarlos.com\/wp-content\/uploads\/2025\/12\/Free-Robots.txt-Generator-768x432.webp 768w\" sizes=\"(max-width: 1280px) 100vw, 1280px\" title=\"\">\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-84b5082 elementor-toc--minimized-on-desktop elementor-widget elementor-widget-table-of-contents\" data-id=\"84b5082\" data-element_type=\"widget\" data-e-type=\"widget\" data-settings=\"{&quot;headings_by_tags&quot;:[&quot;h2&quot;],&quot;exclude_headings_by_selector&quot;:[],&quot;collapse_subitems&quot;:&quot;yes&quot;,&quot;no_headings_message&quot;:&quot;No headings were found on this page.&quot;,&quot;minimized_on&quot;:&quot;desktop&quot;,&quot;marker_view&quot;:&quot;numbers&quot;,&quot;minimize_box&quot;:&quot;yes&quot;,&quot;hierarchical_view&quot;:&quot;yes&quot;,&quot;min_height&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;min_height_tablet&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]},&quot;min_height_mobile&quot;:{&quot;unit&quot;:&quot;px&quot;,&quot;size&quot;:&quot;&quot;,&quot;sizes&quot;:[]}}\" data-widget_type=\"table-of-contents.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div class=\"elementor-toc__header\">\n\t\t\t\t\t\t<div class=\"elementor-toc__header-title\">\n\t\t\t\tDaftar Isi\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t<div class=\"elementor-toc__toggle-button elementor-toc__toggle-button--expand\" role=\"button\" tabindex=\"0\" aria-controls=\"elementor-toc__84b5082\" aria-expanded=\"true\" aria-label=\"Open table of contents\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 448 512\"><path d=\"M207.029 381.476L12.686 187.132c-9.373-9.373-9.373-24.569 0-33.941l22.667-22.667c9.357-9.357 24.522-9.375 33.901-.04L224 284.505l154.745-154.021c9.379-9.335 24.544-9.317 33.901.04l22.667 22.667c9.373 9.373 9.373 24.569 0 33.941L240.971 381.476c-9.373 9.372-24.569 9.372-33.942 0z\"><\/path><\/svg><\/div>\n\t\t\t\t<div class=\"elementor-toc__toggle-button elementor-toc__toggle-button--collapse\" role=\"button\" tabindex=\"0\" aria-controls=\"elementor-toc__84b5082\" aria-expanded=\"true\" aria-label=\"Close table of contents\"><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 448 512\"><path d=\"M240.971 130.524l194.343 194.343c9.373 9.373 9.373 24.569 0 33.941l-22.667 22.667c-9.357 9.357-24.522 9.375-33.901.04L224 227.495 69.255 381.516c-9.379 9.335-24.544 9.317-33.901-.04l-22.667-22.667c-9.373-9.373-9.373-24.569 0-33.941L207.03 130.525c9.372-9.373 24.568-9.373 33.941-.001z\"><\/path><\/svg><\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<div id=\"elementor-toc__84b5082\" class=\"elementor-toc__body elementor-toc__list-items--collapsible\">\n\t\t\t<div class=\"elementor-toc__spinner-container\">\n\t\t\t\t<svg class=\"elementor-toc__spinner eicon-animation-spin e-font-icon-svg e-eicon-loading\" aria-hidden=\"true\" viewBox=\"0 0 1000 1000\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\"><path d=\"M500 975V858C696 858 858 696 858 500S696 142 500 142 142 304 142 500H25C25 237 238 25 500 25S975 237 975 500 763 975 500 975Z\"><\/path><\/svg>\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-1574994 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"1574994\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-89c087c\" data-id=\"89c087c\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-2848f36 elementor-widget elementor-widget-heading\" data-id=\"2848f36\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Generator file robots.txt gratis<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-cd4ce4f elementor-widget elementor-widget-html\" data-id=\"cd4ce4f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<div id=\"robots-generator\" style=\"max-width:950px;font-family:system-ui,sans-serif\">\r\n\r\n  <!-- DIRECTORIES -->\r\n  <h3>Blokir Direktori Sensitif<\/h3>\r\n  <label><input type=\"checkbox\" value=\"\/wp-admin\/\">  \/wp-admin\/ (Dampak CPU tinggi)<\/label><br>  <label><input type=\"checkbox\" value=\"\/wp-includes\/\">  \/wp-includes\/ (Dampak CPU sedang)<\/label><br>  <label><input type=\"checkbox\" value=\"\/cgi-bin\/\">  \/cgi-bin\/ (Dampak CPU rendah)<\/label><br><br>\r\n  <label>\r\n    <input type=\"checkbox\" id=\"allowAjax\">\r\n  Izinkan \/wp-admin\/admin-ajax.php\r\n  <\/label><br><br>\r\n  <!-- SEARCH CRAWLERS -->\r\n  <h3>Perayap Mesin Pencari (Default: Izinkan)<\/h3>\r\n  <small>Pilih perayap untuk memblokirnya<\/small><br><br>\r\n  <div style=\"columns:2\">\r\n    <label><input type=\"checkbox\" data-agent=\"Googlebot\">  Google<\/label><br>    <label><input type=\"checkbox\" data-agent=\"bingbot\">  Bing<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Slurp\">  Yahoo<\/label><br>    <label><input type=\"checkbox\" data-agent=\"DuckDuckBot\">  DuckDuckGo<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Baiduspider\">  Baidu<\/label><br>    <label><input type=\"checkbox\" data-agent=\"YandexBot\">  Yandex<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Applebot\">  Apple<\/label><br>    <label><input type=\"checkbox\" data-agent=\"NaverBot\">  Naver<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Teoma\">  Tanya \/ Teoma<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Gigabot\">  GigaBlast<\/label><br>    <label><input type=\"checkbox\" data-agent=\"msnbot\">  Pencarian MSN<\/label><br>    <label><input type=\"checkbox\" data-agent=\"msnbot-media\">  MSN PicSearch<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Googlebot-Image\">  Google Images<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Googlebot-Mobile\">  Google Seluler<\/label><br>  <\/div>\r\n\r\n  <!-- OTHER KNOWN CRAWLERS -->\r\n  <h3>Perayap SEO &amp; Utilitas (Default: Izinkan)<\/h3>\r\n  <div style=\"columns:2\">\r\n    <label><input type=\"checkbox\" data-agent=\"AhrefsBot\">  Ahrefs<\/label><br>    <label><input type=\"checkbox\" data-agent=\"SemrushBot\">  Semrush<\/label><br>    <label><input type=\"checkbox\" data-agent=\"MJ12bot\">  Megah<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Mozbot\">  Moz<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Screaming Frog SEO Spider\">  Screaming Frog<\/label><br>    <label><input type=\"checkbox\" data-agent=\"CCBot\">  Perayapan Umum<\/label><br>    <label><input type=\"checkbox\" data-agent=\"ia_archiver\">  Alexa \/ Wayback<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Pinterestbot\">  Pinterest<\/label><br>    <label><input type=\"checkbox\" data-agent=\"facebookexternalhit\">  Facebook<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Twitterbot\">  Twitter<\/label><br>  <\/div>\r\n\r\n  <!-- AI CRAWLERS -->\r\n  <h3>Bot Pelatihan Perayap AI &amp; LLM (Default: Izinkan)<\/h3>\r\n  <small>Memblokir ini dapat mengurangi beban server dan penggunaan data<\/small><br><br>\r\n  <div style=\"columns:2\">\r\n    <label><input type=\"checkbox\" data-agent=\"GPTBot\">  OpenAI (GPTBot) - Dampak CPU yang tinggi<\/label><br>    <label><input type=\"checkbox\" data-agent=\"ChatGPT-User\">  Pengguna ChatGPT<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Google-Extended\">  Google AI (SGE \/ Gemini)<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Meta-ExternalAgent\">  Meta AI<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Applebot-Extended\">  Apple AI<\/label><br>    <label><input type=\"checkbox\" data-agent=\"PerplexityBot\">  AI Kebingungan<\/label><br>    <label><input type=\"checkbox\" data-agent=\"ClaudeBot\">  Antropik \/ Claude<\/label><br>    <label><input type=\"checkbox\" data-agent=\"Bytespider\">  TikTok \/ ByteDance AI<\/label><br>  <\/div>\r\n\r\n  <!-- OPTIONS -->\r\n  <h3>Pengaturan Opsional<\/h3>\r\n  <label>Penundaan perayapan (detik)<\/label><br>  <input type=\"number\" id=\"crawlDelay\" min=\"0\"><br><br>\r\n  <label>URL Peta Situs<\/label><br>  <input type=\"url\" id=\"sitemapUrl\" placeholder=\"https:\/\/example.com\/sitemap.xml\" style=\"width:100%\"><br><br>\r\n  <button onclick=\"generateRobots()\" style=\"padding:10px 20px;font-weight:bold\">\r\n  Hasilkan robots.txt\r\n  <\/button>\r\n\r\n  <button onclick=\"downloadRobots()\" style=\"padding:10px 20px;font-weight:bold;margin-left:10px\">\r\n  Unduh robots.txt\r\n  <\/button>\r\n\r\n  <h3>Robot.txt yang dihasilkan<\/h3>\r\n  <textarea id=\"output\" rows=\"20\" style=\"width:100%;font-family:monospace\"><\/textarea>\r\n<\/div>\r\n\r\n<script>\r\nfunction generateRobots() {\r\n  let output = [];\r\n  output.push(\"User-agent: *\");\r\n\r\n  document.querySelectorAll('#robots-generator input[type=\"checkbox\"]:checked').forEach(cb => {\r\n    if (cb.value && !cb.dataset.agent && cb.id !== \"allowAjax\") {\r\n      output.push(\"Disallow: \" + cb.value);\r\n    }\r\n  });\r\n\r\n  if (document.getElementById(\"allowAjax\").checked) {\r\n    output.push(\"Allow: \/wp-admin\/admin-ajax.php\");\r\n  }\r\n\r\n  let delay = document.getElementById(\"crawlDelay\").value;\r\n  if (delay) {\r\n    output.push(\"Crawl-delay: \" + delay);\r\n  }\r\n\r\n  document.querySelectorAll('#robots-generator input[data-agent]:checked').forEach(cb => {\r\n    output.push(\"\");\r\n    output.push(\"User-agent: \" + cb.dataset.agent);\r\n    output.push(\"Disallow: \/\");\r\n  });\r\n\r\n  let sitemap = document.getElementById(\"sitemapUrl\").value;\r\n  if (sitemap) {\r\n    output.push(\"\");\r\n    output.push(\"Sitemap: \" + sitemap);\r\n  }\r\n\r\n  document.getElementById(\"output\").value = output.join(\"\\n\");\r\n}\r\n\r\nfunction downloadRobots() {\r\n  const content = document.getElementById(\"output\").value;\r\n  if (!content) {\r\n    alert(\"Please generate the robots.txt file first.\");\r\n    return;\r\n  }\r\n\r\n  const blob = new Blob([content], { type: \"text\/plain\" });\r\n  const url = URL.createObjectURL(blob);\r\n\r\n  const a = document.createElement(\"a\");\r\n  a.href = url;\r\n  a.download = \"robots.txt\";\r\n  document.body.appendChild(a);\r\n  a.click();\r\n\r\n  document.body.removeChild(a);\r\n  URL.revokeObjectURL(url);\r\n}\r\n<\/script>\r\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-1cc93a8b elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"1cc93a8b\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-10e15af5\" data-id=\"10e15af5\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-42f573c1 elementor-widget elementor-widget-heading\" data-id=\"42f573c1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Apa Itu File robots.txt?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-362f6567 elementor-widget elementor-widget-text-editor\" data-id=\"362f6567\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p data-start=\"1031\" data-end=\"1191\"><strong data-start=\"1033\" data-end=\"1052\">File robots.txt adalah file teks<\/strong> <strong data-start=\"1058\" data-end=\"1071\">yang ditempatkan di<\/strong> <strong data-start=\"1086\" data-end=\"1117\">folder<\/strong> root <strong data-start=\"1086\" data-end=\"1117\">situs web Anda yang memberi tahu<\/strong> <strong data-start=\"1131\" data-end=\"1156\">perayap mesin<\/strong> pencari <strong data-start=\"1131\" data-end=\"1156\">bagaimana cara<\/strong> merayapi <strong data-start=\"1168\" data-end=\"1190\">situs web Anda.<\/strong><\/p><p data-start=\"1193\" data-end=\"1228\">Secara sederhana, <strong data-start=\"1212\" data-end=\"1227\">file robot:<\/strong><\/p><ul data-start=\"1229\" data-end=\"1520\"><li data-start=\"1229\" data-end=\"1309\"><p data-start=\"1231\" data-end=\"1309\">Memberi tahu <strong data-start=\"1237\" data-end=\"1309\">mesin pencari bagian mana dari situs web Anda yang boleh dirayapi<\/strong><\/p><\/li><li data-start=\"1310\" data-end=\"1378\"><p data-start=\"1312\" data-end=\"1378\">Mencegah perayap mengakses <strong data-start=\"1354\" data-end=\"1378\">direktori atau halaman<\/strong> tertentu<\/p><\/li><li data-start=\"1379\" data-end=\"1437\"><p data-start=\"1381\" data-end=\"1437\">Membantu mengelola <strong data-start=\"1394\" data-end=\"1410\">anggaran perayapan,<\/strong> terutama pada situs besar<\/p><\/li><li data-start=\"1438\" data-end=\"1520\"><p data-start=\"1440\" data-end=\"1520\">Mengurangi perayapan yang tidak perlu yang dapat menyebabkan <strong data-start=\"1484\" data-end=\"1505\">duplikasi konten atau<\/strong> beban server<\/p><\/li><\/ul><p data-start=\"1522\" data-end=\"1650\"><strong data-start=\"1526\" data-end=\"1566\">File robots.txt memberi tahu mesin pencari tentang apa yang<\/strong> dapat dan tidak dapat <strong data-start=\"1526\" data-end=\"1566\">mereka<\/strong> akses &#8211; tetapi <strong data-start=\"1608\" data-end=\"1649\">tidak memaksa pengindeksan atau deindeksan.<\/strong><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-22f49dd9 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"22f49dd9\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-4049dd0a\" data-id=\"4049dd0a\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-7ecc3baa elementor-widget elementor-widget-heading\" data-id=\"7ecc3baa\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Mengapa Situs Web Anda Membutuhkan File robots.txt<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-1964306b elementor-widget elementor-widget-text-editor\" data-id=\"1964306b\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p data-start=\"1702\" data-end=\"1851\">Setiap situs web berjalan dengan <strong data-start=\"1726\" data-end=\"1742\">anggaran perayapan,<\/strong> yang berarti mesin pencari seperti <strong data-start=\"1772\" data-end=\"1785\">Googlebot dan Bing<\/strong> <strong data-start=\"1790\" data-end=\"1798\">memiliki<\/strong> <strong data-start=\"1806\" data-end=\"1821\">batas<\/strong> perayapan <strong data-start=\"1806\" data-end=\"1821\">untuk berapa<\/strong> banyak URL yang mereka kunjungi.<\/p><p data-start=\"1853\" data-end=\"1897\">Menggunakan <strong data-start=\"1859\" data-end=\"1883\">robots.txt dengan benar akan memastikan hal itu<\/strong>:<\/p><ul data-start=\"1898\" data-end=\"2085\"><li data-start=\"1898\" data-end=\"1950\"><p data-start=\"1900\" data-end=\"1950\">Mesin telusur <strong data-start=\"1915\" data-end=\"1950\">merayapi dan mengindeks halaman-halaman penting<\/strong><\/p><\/li><li data-start=\"1951\" data-end=\"1979\"><p data-start=\"1953\" data-end=\"1979\">URL bernilai rendah diabaikan<\/p><\/li><li data-start=\"1980\" data-end=\"2033\"><p data-start=\"1982\" data-end=\"2033\"><strong data-start=\"1982\" data-end=\"2003\">URL konten duplikat<\/strong> tidak membuang anggaran perayapan<\/p><\/li><li data-start=\"2034\" data-end=\"2085\"><p data-start=\"2036\" data-end=\"2085\">Sumber daya server tidak terbuang sia-sia oleh bot yang agresif<\/p><\/li><\/ul><p data-start=\"2087\" data-end=\"2171\">Jika digunakan dengan benar, <strong data-start=\"2108\" data-end=\"2135\">robots.txt meningkatkan SEO, kinerja<\/strong>, dan efisiensi perayapan.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-349232d3 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"349232d3\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-70d0e4f6\" data-id=\"70d0e4f6\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-531468e elementor-widget elementor-widget-heading\" data-id=\"531468e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Apa yang Dilakukan Generator Robots.txt Gratis Ini<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-475cabd4 elementor-widget elementor-widget-text-editor\" data-id=\"475cabd4\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p data-start=\"2223\" data-end=\"2272\"><strong data-start=\"2227\" data-end=\"2261\">Generator file robots.txt gratis<\/strong> kami <strong data-start=\"2227\" data-end=\"2261\">membantu Anda<\/strong>:<\/p><ul data-start=\"2274\" data-end=\"2617\"><li data-start=\"2274\" data-end=\"2315\"><p data-start=\"2276\" data-end=\"2315\"><strong data-start=\"2276\" data-end=\"2315\">Hasilkan file robots.txt secara instan<\/strong><\/p><\/li><li data-start=\"2316\" data-end=\"2377\"><p data-start=\"2318\" data-end=\"2377\">Buat <strong data-start=\"2325\" data-end=\"2341\">aturan khusus menggunakan<\/strong> <strong data-start=\"2356\" data-end=\"2377\">sintaks robots.txt<\/strong> yang benar<\/p><\/li><li data-start=\"2378\" data-end=\"2417\"><p data-start=\"2380\" data-end=\"2417\">Menambahkan <strong data-start=\"2384\" data-end=\"2417\">arahan mengizinkan dan melarang<\/strong><\/p><\/li><li data-start=\"2418\" data-end=\"2486\"><p data-start=\"2420\" data-end=\"2486\">Kontrol perayapan untuk <strong data-start=\"2441\" data-end=\"2486\">Googlebot, Bing, dan mesin pencari lainnya<\/strong><\/p><\/li><li data-start=\"2487\" data-end=\"2531\"><p data-start=\"2489\" data-end=\"2531\">Blokir bot pelatihan AI tanpa merusak SEO<\/p><\/li><li data-start=\"2532\" data-end=\"2568\"><p data-start=\"2534\" data-end=\"2568\">Optimalkan <strong data-start=\"2543\" data-end=\"2568\">perayapan dan pengindeksan<\/strong><\/p><\/li><li data-start=\"2569\" data-end=\"2617\"><p data-start=\"2571\" data-end=\"2617\">Buat aturan berdasarkan <strong data-start=\"2600\" data-end=\"2617\">kebutuhan situs web<\/strong> Anda<\/p><\/li><\/ul><p data-start=\"2619\" data-end=\"2719\">Alat ini dibuat oleh <strong data-start=\"2643\" data-end=\"2657\">pakar SEO yang berfokus<\/strong> pada <strong data-start=\"2669\" data-end=\"2701\">perilaku perayapan di dunia nyata,<\/strong> bukan hanya teori.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-35295016 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"35295016\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-1b9866ab\" data-id=\"1b9866ab\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-2ead07d9 elementor-widget elementor-widget-heading\" data-id=\"2ead07d9\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Cara Kerja robots.txt (Penjelasan Singkat)<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-3036d2d3 elementor-widget elementor-widget-text-editor\" data-id=\"3036d2d3\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p data-start=\"2771\" data-end=\"2798\">File robots.txt berisi:<\/p><ul data-start=\"2799\" data-end=\"2970\"><li data-start=\"2799\" data-end=\"2847\"><p data-start=\"2801\" data-end=\"2847\"><strong data-start=\"2801\" data-end=\"2815\">Agen-pengguna (<\/strong> bot mana yang berlaku untuk aturan tersebut)<\/p><\/li><li data-start=\"2848\" data-end=\"2915\"><p data-start=\"2850\" data-end=\"2915\"><strong data-start=\"2850\" data-end=\"2864\">Arahan (<\/strong> aturan seperti <code data-start=\"2877\" data-end=\"2884\">Allow<\/code>, <code data-start=\"2886\" data-end=\"2896\">Disallow<\/code>, atau <code data-start=\"2901\" data-end=\"2914\">Crawl-delay<\/code>)<\/p><\/li><li data-start=\"2916\" data-end=\"2970\"><p data-start=\"2918\" data-end=\"2970\">Referensi opsional seperti <strong data-start=\"2947\" data-end=\"2970\">URL peta situs Anda<\/strong><\/p><\/li><\/ul><p data-start=\"2972\" data-end=\"2980\">Contoh:<\/p><div class=\"contain-inline-size rounded-2xl corner-superellipse\/1.1 relative bg-token-sidebar-surface-primary\"><div class=\"@w-xl\/main:top-9 sticky top-[calc(--spacing(9)+var(--header-height))]\"><div class=\"absolute end-0 bottom-0 flex h-9 items-center pe-2\"><div class=\"bg-token-bg-elevated-secondary text-token-text-secondary flex items-center gap-4 rounded-sm px-2 font-sans text-xs\"> <\/div><\/div><\/div><div class=\"overflow-y-auto p-4\" dir=\"ltr\"><code class=\"whitespace-pre! language-txt\">User-agent: Googlebot\nAllow: \/\n\nUser-agent: *\nDisallow: \/wp-admin\/\n<\/code><\/div><\/div><p data-start=\"3060\" data-end=\"3176\">Setiap <strong data-start=\"3065\" data-end=\"3078\">arahan memberi tahu<\/strong> <strong data-start=\"3087\" data-end=\"3113\">mesin pencari atau<\/strong> perayap <strong data-start=\"3087\" data-end=\"3113\">tertentu<\/strong> bagaimana cara berperilaku ketika mencoba <strong data-start=\"3156\" data-end=\"3175\">merayapi situs Anda.<\/strong><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-75221af3 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"75221af3\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-1f661613\" data-id=\"1f661613\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-4cd0c3f3 elementor-widget elementor-widget-heading\" data-id=\"4cd0c3f3\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Arahan umum robots.txt Dijelaskan<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-2544a7ff elementor-widget elementor-widget-text-editor\" data-id=\"2544a7ff\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h3 data-start=\"3226\" data-end=\"3240\">Agen-pengguna<\/h3><p data-start=\"3241\" data-end=\"3339\">Menentukan <strong data-start=\"3255\" data-end=\"3262\">bot<\/strong> mana yang berlaku untuk aturan <strong data-start=\"3255\" data-end=\"3262\">tersebut<\/strong> (misalnya, <code data-start=\"3290\" data-end=\"3301\">Googlebot<\/code>, <code data-start=\"3303\" data-end=\"3312\">Bingbot<\/code>, atau <code data-start=\"3317\" data-end=\"3320\">*<\/code> untuk semua perayap).<\/p><h3 data-start=\"3341\" data-end=\"3353\">Melarang<\/h3><p data-start=\"3354\" data-end=\"3460\">Gunakan <strong data-start=\"3362\" data-end=\"3384\">arahan disallow untuk<\/strong> mencegah perayap mengakses <strong data-start=\"3422\" data-end=\"3429\">URL,<\/strong> <strong data-start=\"3431\" data-end=\"3441\">folder, atau direktori<\/strong> <strong data-start=\"3446\" data-end=\"3459\">.<\/strong><\/p><h3 data-start=\"3462\" data-end=\"3471\">Izinkan<\/h3><p data-start=\"3472\" data-end=\"3539\">Mengesampingkan aturan disallow untuk URL tertentu yang ingin diindeks.<\/p><h3 data-start=\"3541\" data-end=\"3556\">Penundaan perayapan<\/h3><p data-start=\"3557\" data-end=\"3638\">Mengontrol seberapa cepat perayap harus meminta halaman (berguna untuk server yang sedang dimuat).<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-14c7d353 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"14c7d353\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-56af137c\" data-id=\"56af137c\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-3dc7270 elementor-widget elementor-widget-heading\" data-id=\"3dc7270\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Tempat Mengunggah File robots.txt<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-51ea17c2 elementor-widget elementor-widget-text-editor\" data-id=\"51ea17c2\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p data-start=\"3685\" data-end=\"3728\">Setelah Anda <strong data-start=\"3694\" data-end=\"3717\">menghasilkan robots.txt, Anda harus<\/strong> melakukannya:<\/p><ol data-start=\"3730\" data-end=\"3930\"><li data-start=\"3730\" data-end=\"3777\"><p data-start=\"3733\" data-end=\"3777\">Simpan sebagai <strong data-start=\"3746\" data-end=\"3758\">file txt bernama<\/strong>  <code data-start=\"3765\" data-end=\"3777\">robots.txt<\/code><\/p><\/li><li data-start=\"3778\" data-end=\"3894\"><p data-start=\"3781\" data-end=\"3846\">Unggah berkas robots.txt ke <strong data-start=\"3815\" data-end=\"3846\">folder<\/strong> root <strong data-start=\"3815\" data-end=\"3846\">situs web Anda<\/strong><\/p><div class=\"contain-inline-size rounded-2xl corner-superellipse\/1.1 relative bg-token-sidebar-surface-primary\"><div class=\"@w-xl\/main:top-9 sticky top-[calc(--spacing(9)+var(--header-height))]\"><div class=\"absolute end-0 bottom-0 flex h-9 items-center pe-2\"><div class=\"bg-token-bg-elevated-secondary text-token-text-secondary flex items-center gap-4 rounded-sm px-2 font-sans text-xs\"> <\/div><\/div><\/div><div class=\"overflow-y-auto p-4\" dir=\"ltr\"><code class=\"whitespace-pre!\">https:<span class=\"hljs-comment\">\/\/example.com\/robots.txt<\/span>\n<\/code><\/div><\/div><\/li><li data-start=\"3895\" data-end=\"3930\"><p data-start=\"3898\" data-end=\"3930\">Pastikan dapat diakses oleh publik<\/p><\/li><\/ol><p data-start=\"3932\" data-end=\"3992\">File ini harus berada di <strong data-start=\"3959\" data-end=\"3967\">root,<\/strong> bukan di dalam subfolder.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-4627379b elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"4627379b\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-66ad0721\" data-id=\"66ad0721\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-31db8715 elementor-widget elementor-widget-heading\" data-id=\"31db8715\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Kirimkan File robots.txt Anda ke Google<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-18b44bd6 elementor-widget elementor-widget-text-editor\" data-id=\"18b44bd6\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p data-start=\"4041\" data-end=\"4096\">Setelah Anda <strong data-start=\"4051\" data-end=\"4083\">memperbarui file robots.txt Anda, Anda harus<\/strong> melakukannya:<\/p><ul data-start=\"4098\" data-end=\"4243\"><li data-start=\"4098\" data-end=\"4130\"><p data-start=\"4100\" data-end=\"4130\">Buka <strong data-start=\"4105\" data-end=\"4130\">Konsol Penelusuran Google<\/strong><\/p><\/li><li data-start=\"4131\" data-end=\"4162\"><p data-start=\"4133\" data-end=\"4162\">Gunakan <strong data-start=\"4141\" data-end=\"4162\">penguji robots.txt<\/strong><\/p><\/li><li data-start=\"4163\" data-end=\"4188\"><p data-start=\"4165\" data-end=\"4188\">Mengirim atau menguji file<\/p><\/li><li data-start=\"4189\" data-end=\"4243\"><p data-start=\"4191\" data-end=\"4243\">Verifikasi bahwa <strong data-start=\"4203\" data-end=\"4220\">pencarian Google dapat<\/strong> merayapi URL yang diizinkan<\/p><\/li><\/ul><p data-start=\"4245\" data-end=\"4350\">Mengirimkan <strong data-start=\"4260\" data-end=\"4304\">file robots.txt ke Google Search Console memastikan bahwa<\/strong> Google membaca versi terbaru.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-6f57f64 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"6f57f64\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-0871e50\" data-id=\"0871e50\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-39cf06b elementor-widget elementor-widget-heading\" data-id=\"39cf06b\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">robots.txt vs Pengindeksan (Penting)<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-48af0d8 elementor-widget elementor-widget-text-editor\" data-id=\"48af0d8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p data-start=\"4396\" data-end=\"4413\">Menggunakan robots.txt:<\/p><ul data-start=\"4414\" data-end=\"4576\"><li data-start=\"4414\" data-end=\"4479\"><p data-start=\"4416\" data-end=\"4479\">\u274c TIDAK menjamin halaman akan dihapus dari hasil pencarian<\/p><\/li><li data-start=\"4480\" data-end=\"4519\"><p data-start=\"4482\" data-end=\"4519\">\u2705 Mengontrol <strong data-start=\"4493\" data-end=\"4505\">perayapan,<\/strong> bukan pengindeksan<\/p><\/li><li data-start=\"4520\" data-end=\"4576\"><p data-start=\"4522\" data-end=\"4576\">\u2705 Membantu <strong data-start=\"4530\" data-end=\"4576\">mesin pencari merayapi dan mengindeks secara efisien<\/strong><\/p><\/li><\/ul><p data-start=\"4578\" data-end=\"4642\">Jika sebuah halaman <strong data-start=\"4595\" data-end=\"4627\">tidak muncul dalam hasil pencarian,<\/strong> Anda mungkin perlu:<\/p><ul data-start=\"4643\" data-end=\"4726\"><li data-start=\"4643\" data-end=\"4664\"><p data-start=\"4645\" data-end=\"4664\"><code data-start=\"4645\" data-end=\"4654\">noindex<\/code>  tag meta<\/p><\/li><li data-start=\"4665\" data-end=\"4688\"><p data-start=\"4667\" data-end=\"4688\">URL kanonik yang tepat<\/p><\/li><li data-start=\"4689\" data-end=\"4726\"><p data-start=\"4691\" data-end=\"4726\">Alat penghapusan di <strong data-start=\"4708\" data-end=\"4726\">Konsol Pencarian<\/strong><\/p><\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-097ec62 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"097ec62\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-0994d33\" data-id=\"0994d33\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-b878e97 elementor-widget elementor-widget-heading\" data-id=\"b878e97\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">robots.txt dan Konten Duplikat<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-0dddd26 elementor-widget elementor-widget-text-editor\" data-id=\"0dddd26\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p data-start=\"4770\" data-end=\"4815\">File robots.txt yang dikonfigurasi dengan benar akan membantu:<\/p><ul data-start=\"4816\" data-end=\"4950\"><li data-start=\"4816\" data-end=\"4852\"><p data-start=\"4818\" data-end=\"4852\">Mencegah perayapan parameter URL<\/p><\/li><li data-start=\"4853\" data-end=\"4896\"><p data-start=\"4855\" data-end=\"4896\">Kurangi perayapan pencarian dan saring URL<\/p><\/li><li data-start=\"4897\" data-end=\"4950\"><p data-start=\"4899\" data-end=\"4950\">Hindari pemborosan anggaran perayapan untuk <strong data-start=\"4929\" data-end=\"4950\">konten duplikat<\/strong><\/p><\/li><\/ul><p data-start=\"4952\" data-end=\"5021\">Hal ini terutama penting untuk situs web besar dan toko eCommerce.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-55800a3 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"55800a3\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-4216443\" data-id=\"4216443\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-e064f7e elementor-widget elementor-widget-heading\" data-id=\"e064f7e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Perayap AI, Bot, dan Beban Server<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-473f43b elementor-widget elementor-widget-text-editor\" data-id=\"473f43b\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p data-start=\"5067\" data-end=\"5135\">Situs web modern tidak hanya dirayapi oleh mesin pencari, tetapi juga oleh:<\/p><ul data-start=\"5136\" data-end=\"5187\"><li data-start=\"5136\" data-end=\"5154\"><p data-start=\"5138\" data-end=\"5154\">Bot pelatihan AI<\/p><\/li><li data-start=\"5155\" data-end=\"5165\"><p data-start=\"5157\" data-end=\"5165\">Pengikis<\/p><\/li><li data-start=\"5166\" data-end=\"5187\"><p data-start=\"5168\" data-end=\"5187\">Perayap agresif<\/p><\/li><\/ul><p data-start=\"5189\" data-end=\"5347\"><strong data-start=\"5194\" data-end=\"5218\">Generator robots.txt<\/strong> ini <strong data-start=\"5194\" data-end=\"5218\">membantu memblokir<\/strong> perayap AI sambil memastikan bahwa Google <strong data-start=\"5263\" data-end=\"5298\">dan mesin pencari lainnya masih dapat<\/strong> merayapi dan mengindeks konten Anda dengan benar.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-0d261dc elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"0d261dc\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-1794b11\" data-id=\"1794b11\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-adc3f76 elementor-widget elementor-widget-heading\" data-id=\"adc3f76\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Kapan Anda Harus Menggunakan robots.txt<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-7e526b8 elementor-widget elementor-widget-text-editor\" data-id=\"7e526b8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p data-start=\"5389\" data-end=\"5407\">Gunakan robots.txt jika:<\/p><ul data-start=\"5408\" data-end=\"5643\"><li data-start=\"5408\" data-end=\"5450\"><p data-start=\"5410\" data-end=\"5450\">Bagian dari situs Anda tidak boleh dirayapi<\/p><\/li><li data-start=\"5451\" data-end=\"5489\"><p data-start=\"5453\" data-end=\"5489\">Situs Anda berjalan dengan <strong data-start=\"5473\" data-end=\"5489\">anggaran perayapan<\/strong><\/p><\/li><li data-start=\"5490\" data-end=\"5549\"><p data-start=\"5492\" data-end=\"5549\">Anda ingin mengontrol <strong data-start=\"5516\" data-end=\"5542\">perilaku perayap mesin pencari<\/strong> <\/p><\/li><li data-start=\"5550\" data-end=\"5596\"><p data-start=\"5552\" data-end=\"5596\">Halaman tertentu tidak boleh diakses oleh bot<\/p><\/li><li data-start=\"5597\" data-end=\"5643\"><p data-start=\"5599\" data-end=\"5643\">Anda ingin meningkatkan efisiensi perayapan dan SEO<\/p><\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-0c4940e elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"0c4940e\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-930092a\" data-id=\"930092a\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-83550c3 elementor-widget elementor-widget-heading\" data-id=\"83550c3\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Hasilkan File robots.txt Anda Sekarang<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-b701be9 elementor-widget elementor-widget-text-editor\" data-id=\"b701be9\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p data-start=\"5688\" data-end=\"5735\">Gunakan <strong data-start=\"5696\" data-end=\"5725\">generator robots.txt gratis<\/strong> kami <strong data-start=\"5696\" data-end=\"5725\">di atas untuk<\/strong>:<\/p><ul data-start=\"5737\" data-end=\"5920\"><li data-start=\"5737\" data-end=\"5767\"><p data-start=\"5739\" data-end=\"5767\"><strong data-start=\"5739\" data-end=\"5767\">Membuat file robots.txt<\/strong><\/p><\/li><li data-start=\"5768\" data-end=\"5803\"><p data-start=\"5770\" data-end=\"5803\">Buat aturan berdasarkan situs Anda<\/p><\/li><li data-start=\"5804\" data-end=\"5837\"><p data-start=\"5806\" data-end=\"5837\">Uji file Anda sebelum menerapkan<\/p><\/li><li data-start=\"5838\" data-end=\"5869\"><p data-start=\"5840\" data-end=\"5869\">Meningkatkan perayapan dan pengindeksan<\/p><\/li><li data-start=\"5870\" data-end=\"5920\"><p data-start=\"5872\" data-end=\"5920\">Pastikan Google dan Bing merayapi URL yang tepat<\/p><\/li><\/ul><p data-start=\"5922\" data-end=\"6004\">Alat ini membantu Anda <strong data-start=\"5942\" data-end=\"5981\">menghasilkan file robots.txt yang efektif, dengan aman dan<\/strong> instan.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-e7ce08d elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"e7ce08d\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-64dc156a\" data-id=\"64dc156a\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-45ea0e2a elementor-widget elementor-widget-heading\" data-id=\"45ea0e2a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Pertanyaan Umum<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-2591eca elementor-widget elementor-widget-html\" data-id=\"2591eca\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<script type=\"application\/ld+json\">{\r\n  \"@context\": \"https:\/\/schema.org\",\r\n  \"@type\": \"FAQPage\",\r\n  \"mainEntity\": [\r\n    {\r\n      \"@type\": \"Question\",\r\n      \"name\": \"What is a robots.txt file?\",\r\n      \"acceptedAnswer\": {\r\n        \"@type\": \"Answer\",\r\n        \"text\": \"A robots.txt file is a text file placed in the root folder of your website that tells search engine crawlers which parts of your site they are allowed or not allowed to crawl, based on the robots exclusion protocol.\"\r\n      }\r\n    },\r\n    {\r\n      \"@type\": \"Question\",\r\n      \"name\": \"Does robots.txt affect SEO?\",\r\n      \"acceptedAnswer\": {\r\n        \"@type\": \"Answer\",\r\n        \"text\": \"Yes. robots.txt affects SEO by controlling crawling behavior, crawl budget usage, and which URLs search engines can access. It does not directly control indexing but helps search engines crawl your website efficiently.\"\r\n      }\r\n    },\r\n    {\r\n      \"@type\": \"Question\",\r\n      \"name\": \"Can robots.txt block Google from indexing my website?\",\r\n      \"acceptedAnswer\": {\r\n        \"@type\": \"Answer\",\r\n        \"text\": \"robots.txt can prevent Googlebot from crawling URLs, but it does not guarantee removal from search results. To control indexing, use noindex meta tags or Google Search Console removal tools.\"\r\n      }\r\n    },\r\n    {\r\n      \"@type\": \"Question\",\r\n      \"name\": \"Where should I upload the robots.txt file?\",\r\n      \"acceptedAnswer\": {\r\n        \"@type\": \"Answer\",\r\n        \"text\": \"The robots.txt file must be uploaded to the root folder of your website, for example: https:\/\/example.com\/robots.txt. Search engines will only check this location.\"\r\n      }\r\n    },\r\n    {\r\n      \"@type\": \"Question\",\r\n      \"name\": \"Should I block AI crawlers using robots.txt?\",\r\n      \"acceptedAnswer\": {\r\n        \"@type\": \"Answer\",\r\n        \"text\": \"Blocking AI crawlers using robots.txt can reduce unnecessary crawling and server load. However, enforcement is stronger when combined with Cloudflare WAF or bot protection settings.\"\r\n      }\r\n    },\r\n    {\r\n      \"@type\": \"Question\",\r\n      \"name\": \"How do I test my robots.txt file?\",\r\n      \"acceptedAnswer\": {\r\n        \"@type\": \"Answer\",\r\n        \"text\": \"You can test your robots.txt file using the robots.txt tester inside Google Search Console to verify that Googlebot can crawl important URLs and that disallowed rules are working correctly.\"\r\n      }\r\n    }\r\n  ]\r\n}\r\n<\/script>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<section class=\"elementor-section elementor-inner-section elementor-element elementor-element-293be7ee elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"293be7ee\" data-element_type=\"section\" data-e-type=\"section\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;}\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-inner-column elementor-element elementor-element-43950af2\" data-id=\"43950af2\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-1d21b482 elementor-widget elementor-widget-heading\" data-id=\"1d21b482\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Kesimpulan<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<div class=\"elementor-element elementor-element-5b0fee02 elementor-widget elementor-widget-text-editor\" data-id=\"5b0fee02\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<ul data-start=\"6031\" data-end=\"6268\"><li data-start=\"6031\" data-end=\"6078\"><p data-start=\"6033\" data-end=\"6078\">robots.txt adalah file yang memandu <strong data-start=\"6059\" data-end=\"6078\">perayap<\/strong><\/p><\/li><li data-start=\"6079\" data-end=\"6125\"><p data-start=\"6081\" data-end=\"6125\">Harus mengikuti <strong data-start=\"6104\" data-end=\"6125\">sintaks robots.txt<\/strong> yang benar<\/p><\/li><li data-start=\"6126\" data-end=\"6176\"><p data-start=\"6128\" data-end=\"6176\">Selalu uji perubahan di <strong data-start=\"6151\" data-end=\"6176\">Konsol Penelusuran Google<\/strong><\/p><\/li><li data-start=\"6177\" data-end=\"6268\"><p data-start=\"6179\" data-end=\"6268\">Robots.txt yang dikonfigurasi dengan baik memastikan bahwa <strong data-start=\"6221\" data-end=\"6268\">mesin<\/strong> pencari <strong data-start=\"6221\" data-end=\"6268\">merayapi dan mengindeks apa yang penting<\/strong><\/p><\/li><\/ul><p data-start=\"6270\" data-end=\"6435\">Jika Anda ingin diindeks, mengontrol perilaku perayapan, dan melindungi server Anda, gunakan <strong data-start=\"6347\" data-end=\"6384\">generator robots.txt<\/strong> gratis kami <strong data-start=\"6347\" data-end=\"6384\">untuk membuat file<\/strong> robots.txt <strong data-start=\"6347\" data-end=\"6384\">yang bersih dan aman untuk SEO<\/strong> hari ini.<\/p><p data-start=\"193\" data-end=\"611\">Mengetahui cara <strong data-start=\"208\" data-end=\"229\">membuat file robots.txt dengan benar<\/strong> adalah langkah penting untuk situs web apa pun yang ingin mengontrol perayapan dan meningkatkan <strong data-start=\"312\" data-end=\"332\">SEO Anda.<\/strong> Ketika Anda <strong data-start=\"343\" data-end=\"371\">membuat file robots.txt, Anda mendefinisikan<\/strong> file yang <strong data-start=\"392\" data-end=\"414\">berisi instruksi yang jelas<\/strong> untuk perayap mesin pencari seperti Google, <strong data-start=\"461\" data-end=\"476\">membantu mereka<\/strong> memahami halaman mana di <strong data-start=\"508\" data-end=\"533\">situs web<\/strong> Anda <strong data-start=\"508\" data-end=\"533\">yang harus dirayapi<\/strong> dan bagian mana dari <strong data-start=\"562\" data-end=\"593\">situs web<\/strong> mereka <strong data-start=\"562\" data-end=\"593\">yang perlu dibatasi<\/strong>. <\/p><p data-start=\"613\" data-end=\"960\">Dengan <strong data-start=\"616\" data-end=\"650\">membuat file robots.txt, Anda dapat<\/strong> dengan mudah membuat file yang <strong data-start=\"675\" data-end=\"694\">berisi daftar aturan untuk bot<\/strong>, mengelola perayapan berdasarkan <strong data-start=\"727\" data-end=\"753\">batas<\/strong> perayapan <strong data-start=\"727\" data-end=\"753\">, dan menghindari pemborosan<\/strong> anggaran perayapan untuk URL yang tidak perlu. Saat menambahkan aturan, ingatlah untuk menambahkan <strong data-start=\"838\" data-end=\"863\">garis miring ke depan dan tentukan<\/strong> <strong data-start=\"886\" data-end=\"914\">alamat direktori yang ingin Anda<\/strong> izinkan atau blokir untuk mencegah kesalahan. <\/p><p data-start=\"962\" data-end=\"1254\">Setelah aturan Anda selesai, selalu <strong data-start=\"1000\" data-end=\"1028\">kirimkan file robots.txt ke mesin pencari<\/strong> dan verifikasi menggunakan alat pengujian untuk memastikan perilaku yang tepat. File robots.txt yang terstruktur dengan baik akan memberikan Anda kontrol perayapan yang lebih baik, pengindeksan yang lebih bersih, dan fondasi teknis yang lebih kuat untuk kesuksesan SEO jangka panjang. <\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/main>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Ditulis oleh Gunakan generator robots.txt gratis kami untuk membuat file robots.txt secara instan yang membantu Google dan mesin pencari lainnya merayapi dan mengindeks situs web Anda dengan benar, tanpa membuang-buang anggaran perayapan atau mengekspos bagian sensitif situs Anda. Alat yang mudah digunakan ini menghasilkan file robots.txt yang efektif untuk situs web Anda, mengikuti protokol pengecualian [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":400583,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"elementor_header_footer","meta":{"_acf_changed":false,"footnotes":""},"class_list":["post-401904","page","type-page","status-publish","has-post-thumbnail","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/barrazacarlos.com\/id\/wp-json\/wp\/v2\/pages\/401904","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/barrazacarlos.com\/id\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/barrazacarlos.com\/id\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/barrazacarlos.com\/id\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/barrazacarlos.com\/id\/wp-json\/wp\/v2\/comments?post=401904"}],"version-history":[{"count":0,"href":"https:\/\/barrazacarlos.com\/id\/wp-json\/wp\/v2\/pages\/401904\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/barrazacarlos.com\/id\/wp-json\/wp\/v2\/media\/400583"}],"wp:attachment":[{"href":"https:\/\/barrazacarlos.com\/id\/wp-json\/wp\/v2\/media?parent=401904"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}